Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] AIOps Log Rate Analysis: Improve query generators #171008

Merged
merged 11 commits into from
Nov 16, 2023
Merged

[ML] AIOps Log Rate Analysis: Improve query generators #171008

merged 11 commits into from
Nov 16, 2023

Conversation

walterra
Copy link
Contributor

@walterra walterra commented Nov 10, 2023 •
edited
Loading

Summary

Part of #168459.

Improves the query generation for some parts of log rate analysis and adds jest tests to verify the improvements.
Previously, some query generators were not concise enough with their checks and it could happen that for example range queries for the overall time range were added twice or that the range query for the overall time range was added although there were also range filters added for the inner baseline and deviation time range, for example:

    "query": {
      "bool": {
        "filter": [
...
          {
            "range": {
              "@timestamp": {
                "gte": 1698796800506,
                "lte": 1699568984807,
                "format": "epoch_millis"
              }
            }
          },
          {
            "bool": {
              "should": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": 1699272000000,
                      "lte": 1699344000000,
                      "format": "epoch_millis"
                    }
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": 1698969600000,
                      "lte": 1699185600000,
                      "format": "epoch_millis"
                    }
                  }
                }
              ]
            }
          }
        ],
...

The PR also unifies some mocks for the jest unit tests.

Checklist

@walterra walterra self-assigned this Nov 10, 2023
@walterra walterra force-pushed the 168459-ml-aiops-fix-query-generators branch from 2c307b8 to 84ef32c Compare November 10, 2023 13:00
@walterra walterra mentioned this pull request Nov 10, 2023
Closed
@walterra walterra force-pushed the 168459-ml-aiops-fix-query-generators branch from 89ba1a9 to ab4a63c Compare November 14, 2023 07:33
@walterra walterra added :ml release_note:skip Skip the PR/issue when compiling release notes v8.12.0 Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis labels Nov 14, 2023
@walterra walterra marked this pull request as ready for review November 14, 2023 08:01
@walterra walterra requested a review from a team as a code owner November 14, 2023 08:01
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@peteharverson peteharverson mentioned this pull request Nov 15, 2023
Closed
8 tasks
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 375.6KB 375.7KB +117.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

peteharverson
peteharverson approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and LGTM.

As discussed, would be good to flag that the analysis needs to be rerun when changing the query in the query bar (known issue in #168459) for 8.12.

@qn895
Copy link
Member

qn895 commented Nov 16, 2023

Code LGTM 🎉

@walterra
Copy link
Contributor Author

@peteharverson I'll pick up the issue with the query bar in a follow up since the scope of this PR was server side only.

@walterra walterra merged commit 4d27bfb into elastic:main Nov 16, 2023
27 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Nov 16, 2023
@walterra walterra deleted the 168459-ml-aiops-fix-query-generators branch November 16, 2023 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peteharverson peteharverson peteharverson approved these changes

@qn895 qn895 qn895 approved these changes

@jgowdyelastic jgowdyelastic Awaiting requested review from jgowdyelastic

Assignees

@walterra walterra

Labels
backport:skip This commit does not require backporting Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml release_note:skip Skip the PR/issue when compiling release notes v8.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@walterra @elasticmachine @kibana-ci @qn895 @peteharverson @kibanamachine